Repository navigation
llama: load the engine's Hadamard-rotated Q4_0 files (onebit.hadamard_q4_0) through llama-hadamard - #58
Conversation
…_q4_0) through llama-hadamard The engine's tools/hadamard_q4_0.py writes Q4_0 files whose matmul weights are rotated per 32-element block along the input dimension by the normalized Sylvester Walsh-Hadamard matrix, without signs, and stamps onebit.hadamard_q4_0 = 32. That is the prism.hadamard transform with block_size 32 and sign_mode identity, so llama-hadamard now reads the stamp too: the rotated weights are the file's Q4_0 tensors on a verified matmul path (a Q4_0 lookup table such as token_embd is written in the plain basis). The activation rotation is an ordinary MUL_MAT against H, so the files run on HRX and every other backend, where before only the engine's ROCm build (which rotates inside its quantizers) could run them. - ssm_alpha / ssm_beta join the verified kinds: qwen35 and qwen35moe build both through build_lora_mm, and the engine's files rotate them. - A stamped weight the model does not load (the MTP layer, blk.64, without --mtp) is skipped; prism.hadamard's named weights stay strict. - verify_graph still refuses a folded weight that reaches a matmul without its transform. Qwen3.8-27B-Q4_0-H32 on HRX0, llama-perplexity, wikitext-2 40 x 512 against the BF16 logits: PPL 6.021, KLD 0.0292, same top token 92.5% - the ROCm build's exact int8 path gives 6.008 / 0.0284 / 92.5% (without the rotation HRX gave PPL 550,134). llama-bench on HRX0 (before fork #55): pp512 63.5, pp2048 62.4, tg64 11.65 tok/s. Q4_0 prompt matmuls still take HRX's generic path; #55 routes the K-quants and IQ4_XS only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
left a comment
bong-water-water-bong
left a comment
There was a problem hiding this comment.
Review (PR-Agent duty). Correct, and all of it is in our llama-hadamard.{h,cpp}. Mapping the onebit.hadamard_q4_0 = 32 stamp onto prism.hadamard (block 32, Sylvester, identity signs) is right, and the numbers back it: HRX PPL 6.021 / KLD 0.0292 against ROCm's exact path at 6.008 / 0.0284, where unrotated HRX gave 550,134.
Three notes, none blocking:
- Which weights are rotated. Here it's every Q4_0 tensor that
foldable(name)accepts. The writer side, the engine'stools/hadamard_q4_0.py, decides independently which tensors it rotates. If the two lists ever diverge (a new foldable kind, or the tool skipping one), a weight gets rotated twice or not at all, silently. Worth one engine test that runs the tool on a tiny model and checks its rotated set equals what this loader would fold. Thessm_alpha/ssm_betaaddition is exactly the kind of change that needs both sides. - Never two rotations. ROCmFPX (#7, engine ggml-org#262) rotates activations inside its quantizers for stamped files. This code adds the rotation as a MUL_MAT. They're separate forks today, so nothing doubles, but if llama-hadamard ever lands in ROCmFPX or the ROCm build reads this stamp too, it would. Keep a comment or assert on whichever side merges later.
- Engine follow-up.
servestill refuses stamped files outside--device rocm("is Hadamard-rotated: it runs on --device rocm only"). Once this is pinned, serve can allow HRX. State the speed: tg64 11.65 on HRX vs the ROCm W4A4 path, and pp via #55 once Q4_0 joins its routing.
Mergeable when checks pass.
| "ffn_gate_exps", "ffn_up_exps", "ffn_down_exps", "ffn_gate_up_exps", | ||
| "ffn_gate_shexp", "ffn_up_shexp", "ffn_down_shexp", | ||
| "ssm_out", | ||
| "ssm_out", "ssm_alpha", "ssm_beta", |
There was a problem hiding this comment.
The onebit auto-discovery path (added below at the if (onebit) loop) treats this foldable() kinds list as the single source of truth for which weight kinds tools/hadamard_q4_0.py rotated. Nothing in the onebit GGUF stamp cross-checks that assumption (unlike the general prism.hadamard path, which lists weight_names explicitly). ssm_alpha/ssm_beta were added here, but LLM_ARCH_QWEN3NEXT (already in the verified-arch switch above) uses a different, combined tensor blk.N.ssm_ba.weight (LLM_TENSOR_SSM_BETA_ALPHA, routed through build_lora_mm in src/models/qwen3next.cpp) which is not in this list. If a future version of the Python tool ever rotates that tensor for a qwen3next onebit file, it would be silently excluded from rotations here, and verify_graph() cannot catch it either -- it only flags matmuls on weights it already knows are rotated, so a rotated-but-untracked weight is consumed with no activation transform and no error, producing silently wrong output. Worth either keeping this list in lockstep with the Python tool deliberately (a comment/test tying them together), or having the onebit stamp carry an explicit kind/weight list the way the general path does.
| return; | ||
| } | ||
| const gguf_type kt = gguf_get_kv_type(meta, kid); | ||
| const int64_t stamp = kt == GGUF_TYPE_INT32 ? gguf_get_val_i32(meta, kid) : |
There was a problem hiding this comment.
This re-implements integer-type coercion for the onebit.hadamard_q4_0 stamp instead of reusing gguf_u32() (defined above at line 103), which already covers UINT32/INT32/UINT16/UINT8 for exactly this 'small unsigned stamp' pattern -- it's used for the structurally identical prism.hadamard.block_size key a few lines down. Here only INT32/UINT32 are handled; if the Python writer ever emits this key as UINT8 or UINT16 (plausible for a value as small as 32, and nothing pins the writer to a specific width), stamp falls through to -1 and this throws a confusing 'unsupported onebit.hadamard_q4_0: -1' instead of reading 32. Simpler and more robust as uint32_t stamp32 = 0; if (!gguf_u32(meta, "onebit.hadamard_q4_0", stamp32, false)) return; onebit = stamp32 == 32; if (!onebit) throw ...;.
| uint32_t version = 0; | ||
| gguf_bool(meta, "prism.hadamard.tied_output", tied_output); | ||
| // The engine's tools/hadamard_q4_0.py stamps onebit.hadamard_q4_0 = 32 instead: every Q4_0 weight of | ||
| // the file is rotated by the normalized 32-point Sylvester Walsh-Hadamard matrix per block along |
There was a problem hiding this comment.
This 4-line comment (and its wrapping) runs against this repo's AGENTS.md (linked from CLAUDE.md), which both caps comments ('Keep code comments concise (usually 1-2 lines)') and forbids fixed-width wrapping ('Avoid hard-wrapping it to a fixed column width - that hurts readability' / 'Do NOT split a line into multiple lines mid-sentence, do NOT try to force the line to fit a fixed number of characters'). The four lines here run 105/101/103/86 characters, the classic shape of wrapping prose to a column rather than letting it run long like the rest of this file's comments do. Worth collapsing to 1-2 unwrapped lines.
| if (onebit) { | ||
| block_size = 32; | ||
| transform = "normalized-sylvester-walsh-hadamard"; | ||
| axis = "input-last-dimension"; |
There was a problem hiding this comment.
block_size/transform/axis/sign_mode are hardcoded here purely so the generic validation a few lines below (block_size power-of-two check, transform/axis/sign_mode string checks) passes unconditionally for the onebit path -- they can never fail for these fixed values. That validation is dead weight for the onebit branch; skipping it for the onebit case (or restructuring so onebit doesn't flow through checks it always satisfies) would be simpler than fabricating fields to satisfy them.
8d880f6
into
1bit/hrx-vulkan-patched
The engine's Hadamard-rotated Q4_0 files can now run on HRX, and on every other backend, through
llama-hadamard.Background
The engine's
tools/hadamard_q4_0.pywrites Q4_0 files whose matmul weights are rotated per 32-element input block by the normalized Sylvester Walsh-Hadamard matrix, without signs. It stamps themonebit.hadamard_q4_0 = 32. That is theprism.hadamardtransform with block size 32 and sign mode identity. Until now only the engine's ROCm build, which rotates inside its quantizers, could read them.Changes
load_keysreads the stamp: when a file has noprism.hadamard.versionbut hasonebit.hadamard_q4_0 = 32, the rotated weights are its Q4_0 tensors on a verified matmul path. A Q4_0 lookup table such astoken_embdis written in the plain basis, so it is left out.ssm_alphaandssm_beta. qwen35 and qwen35moe build both throughbuild_lora_mm, and the engine's files rotate them.blk.64without--mtp, is skipped. Namedprism.hadamardweights are still checked strictly.verify_graphstill refuses a folded weight that reaches a matmul without its transform.Results
Qwen3.8-27B-Q4_0-H32,
llama-perplexityon wikitext-2 (40 × 512) against the BF16 logits, andllama-benchon HRX0 (llama-bench measured before #55; this PR is rebased onto it):Q4_0 prompt matmuls still take HRX's generic path, because #55 routes only Q4_K, Q5_K and IQ4_XS. Prompt speed for these files needs that routing extended to Q4_0.
Only
src/llama-hadamard.{cpp,h}change.🤖 Generated with Claude Code